REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  NO.  0704-0188 


Public  Reporting  burden  for  this  collection  ofinformation  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathenng 
and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comment  regarding  this  burden  estimates  or  any  other  aspect  of  this  collection  of 
information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite 
1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188,)  Washington,  DC  20503. _ 


1 .  AGENCY  USE  ONLY  ( Leave  Blank)  2.  REPORT  DATE  3 .  REPORT  TYPE  AND  DATES  COVERED 

08  Jan  2002  FINAL;  01  Sep  1996-31  Aug  2001 


4.  TITLE  AND  SUBTITLE 

Bayesian  Estimation  of  High-Resolution  Imagery  from  Low-Resolution  Video 
Sequences  and  Multisensor  Data  Sets 


5.  FUNDING  NUMBERS 
DAAH04-96-1  -0449 


6.  AUTHOR(S) 

Dr.  Richard  R.  Schultz 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 
University  of  North  Dakota;  Department  of  Electrical  Engineering 
Centennial  Drive;  Upson  II  Room  160 

Grand  Forks,  ND  58202-7165 _ 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

U.  S.  Army  Research  Office 
P.O.  Box  12211 

Research  Triangle  Park,  NC  27709-2211 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSORING /MONITORING 
AGENCY  REPORT  NUMBER 


aa  -c\-  CPs 


11.  SUPPLEMENTARY  NOTES 

The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  be  construed  as  an  official 
Department  of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


1 2  a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT  12  b.  DISTRIBUTION  CODE 

Approved  for  public  release;  distribution  unlimited. 

13.  ABSTRACT  (Maximum  200  words) 

In  modem  spy  movies  such  as  Enemy  of  the  State  and  Patriot  Games ,  scientists  magnify  digital  satellite  imagery  to  determine 
the  identities  of  people  captured  on  surveillance  video.  These  individuals  appear  rather  blocky  at  the  coarsest  resolution  scales 
because  of  undersampling  by  the  image  sensor  array,  but  they  become  remarkably  clear  after  zooming  in  on  a  particular  region-of- 
interest.  Although  real-world  video  enhancement  algorithms  are  not  capable  of  calculating  the  perfect  results  produced  in. 
Hollywood,  additional  details  can  be  extracted  from  an  image  sequence  by  integrating  several  neighboring  frames  that  contain 
subpixel-resolution  scene/object  motion.  The  five  most  important  research  results  obtained  through  this  grant  activity  related  to 
the  Bayesian  estimation  of  high-resolution  imagery  from  low-resolution  digital  video  include:  (1)  nonlinear  filtering  of  subpixel 
motion  vectors  for  improved  super-resolution  video  enhancement;  (2)  estimation  of  subpixel  motion  fields  from  segmented  image 
sequences;  (3)  super-resolution  enhancement  of  compressed  digital  video;  (4)  multiframe  integration  via  the  projective 
transformation  with  automated  block  matching  feature  point  pair  selection;  and  (5)  development  of  a  Windows-based  video 
enhancement  software  tool.  As  a  follow-up  to  these  R&D  activities,  the  PI  has  been  attempting  to  secure  funding  to  market  super¬ 
resolution  enhancement  technologies  to  law  enforcement  and  national  security  agencies. 


14.  SUBJECT  TERMS  15.  NUMBER  OF  PAGES 

Digital  Video,  Enhancement,  High-Resolution,  Image  Compression,  Image  Processing,  15 

Image  Resolution,  Image  Interpolation,  MPEG,  Multiframe  Integration,  Multisensor,  _ 

Reconnaissance,  Sensor  Fusion,  Surveillance,  Super-Resolution,  Upsampling,  Video  Processing  16.  PRICE  CODE 


1 7.  SECURITY  CLASSIFICATION  1 8.  SECURITY  CLASSIFICATION 

OR  REPORT  ON  THIS  PAGE 

_ UNCLASSIFIED _ _ UNCLASSIFIED _ 

NSN  7540-01-280-5500 


20020125  271 


19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

UNCLASSIFIED 


JAN  1  5  200Z 


20.  LIMITATION  OF  ABSTRACT 


Standard  Form  298  (Rev.2-89) 

Prescribed  by  ANSI  Std.  239-1 8 
298-102 


BAYESIAN  ESTIMATION  OF  HIGH-RESOLUTION  IMAGERY  FROM 
LOW-RESOLUTION  VIDEO  SEQUENCES  AND  MULTISENSOR  DATA  SETS 

FINAL  PROGRESS  REPORT 

DR.  RICHARD  R.  SCHULTZ 

08  JAN  2002 

U.S.  ARMY  RESEARCH  OFFICE 

GRANT  NUMBER  DAAH04-96- 1-0449 

UNIVERSITY  OF  NORTH  DAKOTA 

APPROVED  FOR  PUBLIC  RELEASE; 

DISTRIBUTION  UNLIMITED. 

THE  VIEWS,  OPINIONS,  AND/OR  FINDINGS  CONTAINED  IN  THIS  REPORT  ARE  THOSE  OF  THE 
AUTHOR  AND  SHOULD  NOT  BE  CONSTRUED  AS  AN  OFFICIAL  DEPARTMENT  OF  THE  ARMY 
POSITION,  POLICY,  OR  DECISION,  UNLESS  SO  DESIGNATED  BY  OTHER  DOCUMENTATION. 


n 


1 

1.  Foreword 

Intentionally  omitted  (optional). 


2.  Table  of  Contents 

3.  List  of  Appendixes,  Illustrations  and  Tables . 2 

4.  Statement  of  the  Problem  Studied . 3 

4. 1 .  Abstract  from  Standard  Form  298 . 3 

4.2.  Comprehensive  problem  description . 3 

5.  Summary  of  the  Most  Important  Results . 4 

5.1.  Nonlinear  filtering  of  subpixel  motion  vectors  for  improved  super- -resolution 

video  frame  enhancement . 5 

5.2.  Estimation  of  subpixel-resolution  motion  fields  from  segmented  image  sequences . 5 

5.3.  Super-resolution  enhancement  of  compressed  digital  video . 6 

5.4.  Multiframe  integration  via  the  projective  transformation  with  automated  block  matching 

feature  point  pair  selection . 8 

5.5.  Development  of  a  Windows-based  software  tool  for  the  super-resolution  enhancement  of 

surveillance  image  sequences . 8 

5.6.  Super-resolution  enhancement  of  night  vision  image  sequences . 9 

5.7.  Spacecraft  systems  engineering  -  the  initiation  of  a  multidisciplinary  design  project  at  the 

University  of  North  Dakota . 9 

6.  Listing  of  All  Publications  and  Technical  Reports  Supported  Under  this  Grant  or  Contract . 10 

6(a)  Papers  published  in  peer-reviewed  j  oumals . 10 

6(b)  Papers  published  in  non-peer-reviewed  journals  or  in  conference  proceedings . 11 

6(c)  Papers  presented  at  meetings,  but  not  published  in  conference  proceedings . 13 

6(d)  Manuscripts  submitted,  but  not  published . 13 

6(e)  Technical  reports  submitted  to  ARO . 13 

7.  List  of  All  Participating  Scientific  Personnel . 13 

8.  Report  of  Inventions . 14 

9.  Bibliography . 15 

10.  Appendixes . 15 

3.  List  of  Appendixes,  Illustrations  and  Tables 

None. 


2 


r 


4.  Statement  of  the  Problem  Studied 


4.1.  Abstract  from  Standard  Form  298 

In  modem  spy  movies  such  as  Enemy  of  the  State  and  Patriot  Games ,  scientists  magnify  digital 
satellite  imagery  to  determine  the  identities  of  people  captured  on  surveillance  video.  These  individuals 
appear  rather  blocky  at  the  coarsest  resolution  scales  because  of  undersampling  by  the  image  sensor  array, 
but  they  become  remarkably  clear  after  zooming  in  on  a  particular  region-of-interest.  Although  real-world 
video  enhancement  algorithms  are  not  capable  of  calculating  the  perfect  results  produced  in  Hollywood, 
additional  details  can  be  extracted  from  an  image  sequence  by  integrating  several  neighboring  frames  that 
contain  subpixel-resolution  scene/object  motion.  The  five  most  important  research  results  obtained  through 
this  grant  activity  related  to  the  Bayesian  estimation  of  high-resolution  imagery  from  low-resolution  digital 
video  include:  (1)  nonlinear  filtering  of  subpixel  motion  vectors  for  improved  super-resolution  video 
enhancement;  (2)  estimation  of  subpixel  motion  fields  from  segmented  image  sequences;  (3)  super¬ 
resolution  enhancement  of  compressed  digital  video;  (4)  multiframe  integration  via  the  projective 
transformation  with  automated  block  matching  feature  point  pair  selection;  and  (5)  development  of  a 
Windows-based  video  enhancement  software  tool.  As  a  follow-up  to  these  R&D  activities,  the  PI  has  been 
attempting  to  secure  funding  to  market  super-resolution  enhancement  technologies  to  law  enforcement  and 
national  security  agencies. 


4.2.  Comprehensive  problem  description 

In  modem  spy  movies  such  as  Patriot  Games  and  Enemy  of  the  State ,  whenever  an  analyst  magnifies  a 
satellite  image  on  a  computer,  it  appears  as  a  perfect,  highly  detailed  picture.  In  reality,  enlarging  a  digital 
image  by  using  the  magnifying  glass  tool  in  Adobe®  Photoshop®  generally  results  in  a  very  blocky  scene. 
Real-world  video  enhancement  algorithms  are  simply  not  capable  of  calculating  the  perfect  results  produced 
in  Hollywood;  however,  additional  visual  information  can  be  extracted  from  a  digital  image  sequence  by 
temporally  integrating  several  neighboring  frames  to  compute  a  super-resolution  video  still.  Provided  that 
people  and  objects  move  between  the  digital  video  frames,  this  motion  can  be  exploited  to  improve 
definition  and  to  actually  see  details  where  there  were  once  blocky  pixels. 

The  concepts  of  sampling  and  image  resolution  are  extremely  important,  in  the  context  of  capturing  a 
single  digital  picture  using  a  flatbed  scanner  or  a  digital  still  image  camera,  as  well  as  capturing  a  sequence 
of  pictures  using  a  digital  video  (DV)  camera.  The  resulting  digital  imagery  may  be  undersampled ,  in  which 
each  pixel  appears  blocky  when  viewed  close-up.  A  $20  bill  scanned  at  various  resolutions  (dots  per  inch,  or 
dpi)  can  provide  an  intuitive  understanding  of  this  concept.  As  another  example  from  the  remote  sensing 
scientific  community,  the  Landsat  7  satellite  provides  30-meter  resolution  imagery  to  its  end  users.  In 
essence,  this  means  that  each  image  pixel  represents  a  30-meter  by  30-meter  square  region  on  the  Earth’s 
surface.  Obviously,  there  are  a  large  number  of  details  contained  within  a  single  Landsat  7  pixel  that  cannot 
be  observed  from  the  data  directly.  Postprocessing  the  data  using  various  interpolation  methods  can  help  to 
extract  some  additional  details  from  the  digital  imagery. 

The  human  visual  system  appears  to  be  capable  of  temporally  integrating  information  in  a  video 
sequence  in  such  a  way  that  the  perceived  spatial  resolution  of  the  sequence  appears  much  higher  than  the 
actual  spatial  resolution  of  an  individual  frame.  This  effect  is  not  too  surprising,  given  that  neighboring 
frames  within  an  image  sequence  contain  slightly  different,  but  unique,  information.  To  compute  a  super¬ 
resolution  video  still  from  the  original  low-resolution  data,  the  research  conducted  for  this  project  relies  on  a 
method  known  as  Bayesian  maximum  a  posteriori  (MAP)  estimation.  Bayesian  estimation  is  a  stochastic 
regularization  technique  used  to  reformulate  ill-posed  inverse  problems  such  as  image  restoration, 
reconstruction,  and  enhancement  as  well-posed  problems  with  unique  solutions.  An  edge-preserving 
Markov  random  field  prior  density  is  assumed  for  spatial  images  that  accurately  models  piece-wise  smooth 
data,  and  a  Gaussian  density  is  assumed  for  the  modeling  error  and  noise.  The  Bayesian  estimation 
technique  results  in  a  highly-computational,  iterative  optimization  problem  that  can  be  solved  numerically 
using  custom  software.  A  wide  variety  of  simulations  have  shown  that  this  nonlinear  resolution 
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enhancement  technique  results  in  image  estimates  which  are  visually  and  quantitatively  superior  to  linear 
estimates  generated  by  least  squares  methods. 

Developing  an  accurate  video  observation  model  for  the  image  sequence  data  is  perhaps  the  most 
important  component  of  any  super-resolution  video  enhancement  algorithm.  In  modeling  digital  video 
sequences,  the  motion  between  frames  must  be  estimated  so  that  temporal  correlations  along  object 
trajectories  can  be  incorporated  in  the  model.  A  number  of  projects  supported  by  this  grant  are  concerned 
with  the  accurate  estimation  of  subpixel  motion  vectors  for  super-resolution  video  enhancement. 
Furthermore,  compressed  data  is  becoming  the  standard  in  video  communications;  thus,  several  projects 
involve  the  super-resolution  enhancement  of  compressed  video,  specifically  the  MPEG-1  and  MPEG-2 
formats.  Unfortunately,  image  and  video  enhancement  are  extremely  costly  from  a  computational 
perspective,  such  that  a  great  deal  of  processing  time  is  required  to  generate  an  image  estimate.  For  this 
reason,  a  “quick-and-dirty”  method  of  multiframe  integration  is  proposed,  using  the  projective  camera  model 
for  global  scene  transfonnations.  An  interesting  byproduct  of  this  work  surrounds  the  automated  registration 
of  similar  and  dissimilar  sensor  channels  using  block  matching  feature  point  pair  selection.  Finally,  to  take 
super-resolution  enhancement  technology  from  the  research  laboratory  to  the  desktops  of  law  enforcement 
agencies,  a  user-friendly  system  must  be  developed.  An  effort  to  develop  a  Windows-based  software  tool  for 
the  super-resolution  enhancement  of  surveillance  video  sequences  is  described  in  this  report. 

These  research  results  are  particularly  useful  for  cleaning  up  surveillance  and  reconnaissance  image 
sequences.  For  instance,  after  a  crime  takes  place,  it  is  often  difficult  to  obtain  an  adequate  picture  of  the 
suspects  from  the  surveillance  video.  With  super-resolution  video  enhancement,  multiple  video  frames  can 
be  combined  to  extract  a  high-resolution  image  of  the  suspects  and  their  distinguishing  features,  which  in 
turn  helps  law  enforcement  agents  identify  the  perpetrators.  Quite  obviously,  there  are  a  number  of  national 
security-related  applications  on  the  horizon. 

Dr.  Harry  Nyquist,  one  of  the  pioneers  of  modern-day  telecommunications  technology  and  a  graduate 
of  the  University  of  North  Dakota  (BSEE  1914;  MSEE  1915),  originally  developed  the  sampling  theorem, 
one  of  the  most  significant  discoveries  in  telecommunications  and  signal  processing.  This  theorem  dictates 
the  minimum  sampling  frequency  necessary  for  the  perfect  reconstruction  of  a  continuous-time  signal  from 
its  discrete-time  samples.  Because  of  the  massive  increase  in  desktop  computing  power  during  this  past 
decade,  we  are  just  now  beginning  to  utilize  and  advance  Dr.  Nyquist’ s  theories  in  the  digital  image  and 
video  processing  product  development  arena. 


5.  Summary  of  the  Most  Important  Results 

The  five  most  important  research  and  development  projects  related  to  the  Bayesian  estimation  of  high- 
resolution  imagery  from  low-resolution  image  sequences  conducted  throughout  the  duration  of  this  grant  include: 

1 .  Nonlinear  filtering  of  subpixel  motion  vectors  for  improved  super-resolution  video  frame  enhancement; 

2.  Estimation  of  subpixel -resolution  motion  fields  from  segmented  image  sequences; 

3.  Super-resolution  enhancement  of  compressed  digital  video; 

4.  Multiframe  integration  via  the  projective  transformation  with  automated  block  matching  feature  point  pair 
selection;  and 

5.  Development  of  a  Windows-based  software  tool  for  the  super-resolution  enhancement  of  surveillance  image 
sequences. 

In  addition,  two  other  projects  will  be  described,  since  this  ARO  support  was  extremely  vital  in  initiating  the 
following  ongoing  research  efforts: 

1 .  Super-resolution  enhancement  of  night  vision  image  sequences;  and 

2.  Spacecraft  systems  engineering  -  the  initiation  of  a  multidisciplinary  design  project  at  the  University  of  North 
Dakota. 
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Each  of  these  projects  will  be  summarized,  with  all  relevant  publications/presentations  listed. 

5.1 .  Nonlinear  filtering  of  subpixel  motion  vectors  for  improved  super-resolution  video  frame 
enhancement 

Super-resolution  enhancement  algorithms  are  used  to  estimate  a  high-resolution  video  still  (HRVS) 
from  several  low-resolution  frames,  provided  that  entire  scenes  or  individual  objects  within  a  digital  video 
sequence  move  in  subpixel  displacements.  This  requires  the  estimation  of  subpixel-resolution  motion  vectors 
with  respect  to  the  target  picture  undergoing  resolution  enhancement,  prior  to  computational  video  frame 
integration.  However,  estimating  accurate  subpixel  motion  vectors  between  two  low-resolution,  noisy  video 
frames  has  proven  to  be  a  formidable  challenge.  Applying  block  matching  or  optical  flow  estimation  to 
upsampled  frames  from  the  image  sequence  generally  results  in  relatively  poor  subpixel  motion  vectors,  and 
consequently  inaccurate  regions  within  a  particular  super-resolution  enhanced  video  still.  To  improve  the 
visual  quality  of  the  enhanced  frame,  the  motion  vector  estimates  can  be  refined  through  the  application  of 
various  nonlinear  filters,  with  the  most  promising  results  obtained  through  a-trimmed  mean  filtering. 
Nonlinear  filtering  eliminates  most  of  the  outlier  motion  estimation  errors,  resulting  in  a  more  consistent  set 
of  subpixel  motion  vectors.  Consequently,  an  improvement  in  visual  quality  is  achievable  in  the 
corresponding  HRVS  estimate.  Through  simulations,  it  is  shown  that  nonlinear  motion  vector  filtering  is 
more  appropriate  than  linear  finite  impulse  response  filtering  for  improving  the  visual  quality  of  the  super- 
resolved  frames. 

The  following  conference  publication/presentation  based  on  this  research  was  supported  in  part  by  this 
ARO  grant: 

Richard  R.  Schultz,  “Nonlinear  Filtering  of  Subpixel  Motion  Vectors  for  Improved  Super-Resolution 
Video  Frame  Enhancement.”  Invited  paper  in  Proceedings  of  the  2001 IEEE-EURASIP  Workshop  on 
Nonlinear  Signal  and  Image  Processing  (on  CD-ROM),  Special  Session  on  Nonlinear  Methods  for  Signal 
Resolution  Enhancement,  Baltimore,  MD,  3-6  Jun  2001.  Presented  at  the  workshop  by  Richard  R.  Schultz. 


5.2.  Estimation  of  subpixel-resolution  motion  fields  from  segmented  image  sequences 

Super-resolution  enhancement  algorithms  are  used  to  estimate  a  high-resolution  video  still  (HRVS) 
from  several  low-resolution  frames,  provided  that  objects  within  the  image  sequence  move  with  subpixel 
increments.  However,  estimating  an  accurate  subpixel-resolution  motion  field  between  two  low-resolution, 
noisy  video  frames  has  proven  to  be  a  formidable  challenge.  Upsampling  the  image  sequence  frames 
followed  by  the  application  of  block  matching,  optical  flow  estimation,  or  Bayesian  motion  estimation  results 
in  relatively  poor  subpixel-resolution  motion  fields,  and  consequently  inaccurate  regions  within  the  super¬ 
resolution  enhanced  video  still.  This  is  particularly  true  for  large  interpolation  factors  (greater  than  or  equal 
to  4).  To  improve  the  quality  of  the  subpixel  motion  fields  and  the  corresponding  HRVS,  motion  can  be 
estimated  for  each  object  within  a  segmented  image  sequence.  First,  a  reference  video  frame  is  segmented 
into  its  constituent  objects,  and  a  mask  is  generated  for  each  object  that  represents  its  spatial  location.  As 
described  previously,  subpixel-resolution  motion  estimation  is  then  conducted  by  video  frame  upsampling 
followed  by  the  application  of  a  motion  estimation  algorithm.  Finally,  the  motion  vectors  are  averaged  over 
the  region  of  each  mask  by  applying  an  a-trimmed  mean  filter  to  the  horizontal  and  vertical  components 
separately.  Since  each  object  moves  as  a  single  entity,  averaging  the  displacement  vectors  in  this  manner 
eliminates  many  of  the  motion  estimation  errors  and  results  in  much  more  consistent  subpixel  motion  fields. 
A  substantial  improvement  is  also  visible  within  particular  regions  of  the  HRVS  estimates.  Subpixel- 
resolution  motion  fields  and  HRVS  estimates  are  computed  for  interpolation  factors  of  2,  4,  8,  and  16,  to 
examine  the  benefits  of  object  segmentation  and  motion  field  averaging. 
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The  following  conference  publication/presentation  based  on  this  research  was  supported  in  part  by  this 
ARO  grant: 

Richard  R.  Schultz  and  Robert  L.  Stevenson,  “Estimation  of  Subpixel-Resolution  Motion  Fields  from 
Segmented  Image  Sequences.”  In  Proceedings  of  SPIE  -  Sensor  Fusion:  Architectures ,  Algorithms ,  and 
Applications  II  (AeroSense  ‘98),  volume  3376,  pages  90-101,  Orlando,  FL,  13-17  Apr  1998.  Presented  at  the 
conference  by  Richard  R.  Schultz. 


5.3.  Super-resolution  enhancement  of  compressed  digital  video 

As  a  preprocessing  step  to  the  analysis  of  individual  video  frames,  super-resolution  enhancement  can  be 
used  to  extract  additional  spatial  detail  from  an  image  sequence  through  the  temporal  integration  of  multiple 
frames.  Due  to  the  intra-coded  macroblocks  contained  within  an  MPEG  image  sequence  and  the  overlap  of 
coded,  motion-compensated  macroblocks  between  frames,  it  is  possible  to  perform  super-resolution 
enhancement  on  compressed  video  by  temporally  integrating  a  short  subsequence  of  decoded  pictures.  The 
intra-coded  I-picture  is  a  logical  candidate  for  enhancement,  since  it  contains  the  most  unique  information 
among  the  I-,  P-,  and  B-picture  types.  For  an  MPEG-1  image  sequence  coded  in  the  B-B-I-B-B-P-B-B-P 
display  order,  the  I-picture  may  be  enhanced  using  a  maximum  of  six  integrable  decoded  frames  (B-B-I-B-B- 
P),  if  only  pictures  possessing  encoded  half-pel  motion  vectors  estimated  with  respect  to  the  I-picture  are 
considered.  P-  and  B-pictures  can  also  be  enhanced  using  neighboring  frames,  but  different  pictures  may  be 
integrated.  A  maximum  of  seven  decoded  frames  (I-B-B-P-B-B-P)  can  be  integrated  to  enhance  the  first  P- 
picture  following  an  I-picture,  while  a  maximum  of  three  decoded  frames  (I-B-X-P  or  I-X-B-P)  are  integrable 
in  the  super-resolution  enhancement  of  a  B-picture.  To  calculate  a  Bayesian  high-resolution  video  still 
(HRVS)  estimate  with  an  upsampling  factor  of  IV  pixels,  l/(lY)-th  pel  resolution  motion  vectors  are  required 
for  all  frames  with  respect  to  the  picture  undergoing  enhancement.  Adequate  upsampled  displacements  can 
be  estimated  efficiently  by  utilizing  the  half-pel  MPEG-encoded  motion  vectors  as  initial  conditions  to  a 
reduced-search  block  matching  algorithm.  Improvements  in  visual  quality  can  be  quite  impressive,  provided 
that  global  frame  transformations  such  as  those  produced  by  a  camera  pan  are  present  within  the  sequence. 

Embedding  the  Bayesian  HRVS  algorithm  within  an  MPEG-1  decoder  allows  for  the  super-resolution 
enhancement  of  any  frame  within  a  compressed  image  sequence,  although  many  empirical  limitations  have 
been  discovered  which  affect  the  quality  of  the  video  stills.  As  a  broad  generalization,  any  intra-coded 
macroblocks  that  appear  in  a  frame  are  capable  of  improving  the  visual  appearance  of  the  high-resolution 
video  still  far  more  than  nonintra-coded  macroblocks.  This  is  intuitively  obvious,  since  macroblocks  that  are 
only  motion-compensated  do  not  contribute  additional  information  to  the  sequence.  Experimentally,  it  has 
been  found  that  integrating  more  than  approximately  four  pictures  from  an  arbitrary  MPEG-1  sequence  does 
not  improve  the  quality  of  a  super-resolution  enhanced  video  still.  A  law  of  diminishing  returns  takes  effect; 
Le.f  as  additional  pictures  are  integrated,  more  and  more  motion  estimation  errors  leak  into  the  enhanced 
frame,  further  degrading  the  image.  Mosquito  noise  seems  to  be  quite  prevalent  in  the  super-resolution 
enhancement  of  compressed  video,  apparently  due  to  the  integration  of  macroblocks  compensated  using 
inaccurate  motion  vectors  estimated  in  the  MPEG-1  encoder,  as  well  as  the  integration  of  redundant 
information.  For  example,  enhancing  an  I-picture  through  integration  with  its  following  P-picture  usually 
results  in  a  low-quality  video  still,  since  the  P-picture  consists  primarily  of  forward-compensated  macroblocks 
from  the  I-picture,  and  thus  identical  pixels  between  the  two  frames  must  be  combined.  Counterintuitively, 
the  integration  of  I-  and  B-pictures  provides  higher-quality  video  stills  than  I-P  integrations,  most  likely  since 
the  coded  compensation  differences  in  B-pictures  are  more  significant  than  those  present  in  P -pictures.  By 
understanding  the  errors  that  can  occur  in  the  super-resolution  enhancement  of  compressed  video,  it  is 
possible  to  tune  the  MPEG-1  encoder  to  generate  a  video  bitstream  specifically  tailored  for  high-quality 
super-resolution  enhancement  within  the  decoder. 

Since  the  MPEG  encoder  uses  block  matching  to  compute  motion  vectors,  the  half-pel  displacements 
estimated  between  the  I-,  P-,  and  B-pictures  for  a  particular  sequence  can  be  obtained  from  an  MPEG  decoder 
directly.  These  MPEG  half-pel  motion  vectors  serve  as  an  initial  condition  to  a  reduced-search  block 
matching  algorithm  that  calculates  the  l/(1V)-th  pel  motion  vectors  required  for  super-resolution 
enhancement.  To  estimate  the  displacements,  the  reference  decoded  I-picture  and  a  second  integrable  frame 
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are  first  upsampled  by  a  factor  of  Tr  using  cubic  B-spline  interpolation.  Reduced-search  block  matching  is 
then  applied  to  these  upsampled  pictures,  in  which  a  relatively  small  search  area  is  centered  on  an  MPEG 
vector  upsampled  by  a  factor  of  T r! 2.  This  dramatically  reduces  the  amount  of  computation  time  required  to 
calculate  the  motion  vectors  when  compared  to  estimating  the  subpixel-resolution  displacements  using  full- 
search  block  matching,  with  little  apparent  degradation  in  the  super-resolution  enhanced  frame. 

Simulations  were  conducted,  in  which  six  frames  (B-B-I-B-B-P)  were  extracted  from  an  MPEG-1 
sequence  coded  at  a  compression  ratio  of  approximately  10:1.  These  frames  were  first  decoded  and  then 
integrated  to  compute  the  video  still  image.  In  general,  it  was  found  that  I-picture  enhancement  improves 
with  the  integration  of  each  additional  B-picture,  but  deteriorates  with  each  additional  P-picture. 

Further  research  was  also  conducted  on  the  super-resolution  enhancement  of  MPEG-2,  the  coding 
standard  used  in  high-definition  television  (HDTV),  digital  video  disk  (DVD),  and  virtually  all  other 
commercial  digital  video  products.  A  great  need  exists  to  improve  the  results  of  MPEG-2  enhancement,  and 
the  most  likely  area  to  make  inroads  on  this  task  involves  tuning  the  MPEG  encoder  for  optimal  super¬ 
resolution  decoding.  Compressed  video  contains  little  data  redundancy,  while  the  success  of  any  super¬ 
resolution  video  enhancement  algorithm  is  predicated  on  subpixel-resolution  overlap  (redundancy)  of  moving 
objects  from  frame-to-ffame.  By  understanding  the  source  of  errors  that  can  occur  in  the  super-resolution 
enhancement  of  compressed  video,  it  may  be  possible  to  tune  the  MPEG  encoder  to  generate  a  video 
bitstream  specifically  tailored  for  high-quality  super-resolution  enhancement  within  the  decoder.  In  essence, 
the  MPEG  encoder  should  be  optimized  to  (1)  increase  the  bitrate,  since  a  low  compression  ratio  inherently 
results  in  a  coded  sequence  with  more  information  than  a  high  compression  ratio;  (2)  turn  on  progressive 
scanning,  in  contrast  to  interlaced  scanning  which  contains  less  data;  (3)  estimate  motion  vectors  as  accurately 
as  possible;  (4)  place  additional  intra-coded  macroblocks  in  regions  of  high  motion,  since  intra-coded 
macroblocks  contain  the  most  infonnation  content,  and  the  super-resolution  enhancement  of  high  activity 
regions  is  potentially  fruitful;  and  (5)  place  additional  intra-coded  macroblocks  along  spatial  edges  within 
each  picture,  since  super-resolution  enhancement  along  discontinuities  serves  to  dealias  the  video  still. 


The  following  conference  publications/presentations  based  on  this  research  were  supported  in  part  by 
this  ARO  grant: 

Kyle  J.  Erickson  and  Richard  R.  Schultz,  “MPEG-1  Super-Resolution  Decoding  for  the  Analysis  of 
Video  Still  Images.”  In  Proceedings  of  the  2000  IEEE  Southwest  Symposium  on  Image  Analysis  and 
Interpretation ,  pages  13-17,  Austin,  TX,  2-4  Apr  2000.  Presented  at  the  conference  by  Richard  R.  Schultz. 

Debin  Chen  and  Richard  R.  Schultz,  “Extraction  of  High-Resolution  Frames  from  MPEG  Image 
Sequences.”  In  Proceedings  of  the  1998  IEEE  International  Conference  on  Image  Processing  (on  CD- 
ROM),  Chicago,  IL,  4-7  Oct  1998.  Presented  at  the  conference  by  Richard  R.  Schultz. 


A  patent  application  based  on  the  enhancement  of  MPEG- 1  and  MPEG-2  compressed  video  was  filed 
on  12  May  2000.  This  invention  is  based  on  research  conducted  by  Kyle  J.  Erickson  for  his  master’s  thesis, 
entitled  “Super-Resolution  Enhancement  of  MPEG  Compressed  Video,”  defended  on  10  January  2000. 
Important  information  regarding  the  patent  filing  is  provided  as  follows: 


Patent  Title: 
Inventors: 
Assignee: 
Filing  Date: 
Serial  Number: 
Law  Firm: 
Attorney: 


“Super-Resolution  Enhancement  of  Compressed  Video  Data” 

Richard  R.  Schultz  and  Kyle  J.  Erickson 

University  of  North  Dakota 

12  May  2000 

60/133,765 

Shook,  Hardy  &  Bacon  L.L.P.  (Kansas  City,  MO) 

Mr.  Michael  J.  Gross 
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5.4.  Multiframe  integration  via  the  projective  transformation  with  automated  block  matching 
feature  point  pair  selection 

A  subpixel-resolution  image  registration  algorithm  based  on  the  nonlinear  projective  transfonnation 
model  is  proposed  to  account  for  camera  translation,  rotation,  zoom,  pan,  and  tilt.  Typically,  parameter 
estimation  techniques  for  transformation  models  require  the  user  to  manually  select  feature  points  between  the 
images  undergoing  registration.  In  this  research,  block  matching  is  used  to  automatically  select  correlated 
feature  point  pairs  between  two  images,  and  these  features  are  used  to  calculate  an  iterative  least  squares 
solution  for  the  projective  transfonnation  parameters.  Since  block  matching  is  capable  of  estimating  accurate 
translation  motion  vectors  only  in  discontinuous  edge  regions,  inaccurate  feature  point  pairs  are  statistically 
eliminated  prior  to  computing  the  least  squares  parameter  estimate.  Convergence  of  the  projective 
transformation  model  estimation  algorithm  is  generally  achieved  in  several  iterations.  After  subpixel- 
resolution  image  registration,  a  high-resolution  video  still  may  be  computed  by  integrating  the  registered 
pixels  from  a  short  sequence  of  low-resolution  image  sequence  frames. 

Empirically,  it  was  determined  that  integrating  more  than  approximately  five  raw  (uncompressed)  video 
frames  does  not  significantly  improve  the  spatial  resolution  of  a  particular  frame,  seemingly  due  to  a  law  of 
diminishing  returns.  The  quality  of  the  enhanced  imagery  generated  by  this  “quick-and-dirty”  multiframe 
integration  technique  is  certainly  not  as  high  as  that  obtained  by  Bayesian  super-resolution  video 
enhancement;  however,  this  algorithm  does  not  require  nearly  as  many  computational  resources  as  Bayesian 
super-resolution  enhancement,  and  an  estimate  can  be  calculated  in  several  minutes  rather  than  several  hours. 
Moreover,  this  technique  is  effective  in  automatically  registering  channels  acquired  by  dissimilar  sensors, 
assisting  in  the  fusion  of  data  acquired  by  multiple  sensors. 


The  following  conference  publications/presentations  based  on  this  research  were  supported  in  part  by 
this  ARO  grant: 

Richard  R.  Schultz  and  Mark  G.  Alford,  “Automated  Image  Registration  Using  the  Projective 
Transformation  Model  and  Block  Matching  Feature  Point  Pair  Selection.”  In  Proceedings  ofSPIE  - 
Sensor  Fusion:  Architectures,  Algorithms ,  and  Applications  III  (Aero  Sense  ‘99)y  volume  3719,  pages  MO- 
151,  Orlando,  FL,  7-9  Apr  1999.  Presented  at  the  conference  by  Richard  R.  Schultz. 

Richard  R.  Schultz  and  Mark  G.  Alford,  “Multiframe  Integration  via  the  Projective  Transformation 
with  Automated  Block  Matching  Feature  Point  Selection.”  In  Proceedings  of  the  1999  IEEE 
International  Conference  on  Acoustics,  Speech,  and  Signal  Processing  (on  CD-ROM),  Phoenix,  AZ,  15-19 
Mar  1999.  Presented  at  the  conference  by  Richard  R.  Schultz. 


5.5.  Development  of  a  Windows-based  software  tool  for  the  super-resolution  enhancement  of 
surveillance  image  sequences 

With  the  intent  of  taking  super-resolution  enhancement  algorithms  from  the  research  laboratory  to  the 
desktops  of  law  enforcement  agents,  software  designed  for  execution  on  a  UNIX  workstation  has  been  ported 
to  a  personal  computer  running  the  Microsoft  Windows  NT  operating  system.  To  substantially  reduce  the 
number  of  computations  required  by  the  Bayesian  high-resolution  video  stills  (HRVS)  algorithm,  this 
technique  can  be  applied  to  only  critical  regions  within  an  image  sequence.  The  user-friendly  software 
package  may  be  used  by  security,  law  enforcement,  and  military  personnel  to  perform  image  sequence 
enhancement  using  their  standard  office  computers  without  having  to  invest  in  high-end  engineering 
workstations.  Timing  studies  demonstrate  that  the  Mirage  video  enhancement  toolkit  may  outperform  its 
UNIX  ancestor  on  comparably-priced  machines. 

Because  of  the  need  for  user-friendly  super-resolution  image  sequence  enhancement  tools  by  forensic 
analysis  laboratories,  several  research  and  development  efforts  conducted  under  the  auspices  of  this  ARO 
grant  are  being  considered  for  technology  transfer.  Dr.  Richard  R.  Schultz  founded  Multimedia  Forensics  in 
2001,  a  North  Dakota  sole  proprietorship  specializing  in  software  development  for  the  digital  video 
surveillance  community.  The  intent  is  to  further  develop  super-resolution  enhancement  technologies  for  law 
enforcement  and  national  security  agencies  within  the  public  and  private  sectors,  with  development  efforts 
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surrounding  the  creation  of  video  forensic  plug-ins  for  the  Adobe®  Premiere®  nonlinear  digital  video  editing 
software  package.  The  Principal  Investigator  is  currently  searching  for  a  corporate  partner  to  assist  in  the 
commercialization  of  relevant  super-resolution  enhancement  technologies. 


The  following  conference  publication/presentation  based  on  this  research  was  supported  in  part  by  this 
ARO  grant: 

Kyle  J.  Erickson  and  Richard  R.  Schultz,  “A  Windows-Based  Software  Tool  for  the  Super-Resolution 
Enhancement  of  Surveillance  Image  Sequences.”  In  Proceedings  of  the  1998  Midwest  Symposium  on 
Circuits  and  Systems ,  pages  379-382,  Notre  Dame,  IN,  9-12  August  1998.  Presented  at  the  conference  by 
Kyle  J.  Erickson. 


5.6.  Super-resolution  enhancement  of  night  vision  image  sequences 

Extensive  prior  research  has  been  conducted  on  the  Bayesian  super-resolution  enhancement  of  visible 
light  image  sequences.  For  night  vision  applications,  super-resolution  enhancement  must  be  applied  to  image 
sequences  with  frames  acquired  in  the  thermal  infrared  region  of  the  electromagnetic  spectrum.  Most  of  these 
applications  are  DoD-related,  and  they  include  the  enhancement  of  infrared  reconnaissance  and  surveillance 
imagery,  as  well  as  the  enhancement  of  infrared  imagery  for  automated  target  tracking  and  real-time 
multisensor  fusion.  Several  computational  simulations  have  been  performed,  which  visually  depict  how  well 
the  proposed  technique  enhances  short  night  vision  image  sequences  consisting  of  two-to-five  frames.  This 
technique  is  capable  of  increasing  resolution  by  factors  up  to  16x,  although  video  stills  appear  somewhat 
blocky  at  higher  magnification  factors. 

The  following  conference  publication/presentation  based  on  this  research  was  supported  in  part  by  this 
ARO  grant: 

Darryl  Sale  and  Richard  R.  Schultz,  “Super-Resolution  Enhancement  of  Night  Vision  Image 
Sequences.”  In  Proceedings  of  the  IEEE  Systems,  Man ,  and  Cybernetics  Society  2000  Meeting  (on  CD- 
ROM),  Nashville,  TN,  8-1 1  Oct  2000.  Presented  at  the  conference  by  Richard  R.  Schultz. 


5.7.  Spacecraft  systems  engineering  -  the  initiation  of  a  multidisciplinary  design  project  at  the 
University  of  North  Dakota 

During  this  past  year,  several  departments  at  the  University  of  North  Dakota  have  been  focusing  on  the 
design  of  inexpensive  spacecraft  for  atmospheric  studies  and  remote  sensing.  This  multidisciplinary  design 
project  emphasizes  the  systems  engineering  approach,  in  which  extensive  design  documentation  is  created 
prior  to  the  build  of  subsystems  and  a  multilevel  system  integration  and  test.  Fourteen  undergraduate  and 
three  graduate  students  designed  and  tested  the  second  build  of  a  spacecraft,  known  as  Scorpio  II,  launched 
using  a  latex  weather  balloon  in  May  2001 .  This  project  was  conducted  to  teach  the  students  about  systems 
engineering  design  principles,  as  well  as  to  collect  and  transmit  real-time  sensor  data  from  the  atmosphere  and 
digital  images  of  the  Earth’s  surface.  The  telecommunications  subsystem  is  responsible  for  collecting  sensor 
readings,  acquiring  images  from  a  digital  camera,  and  packetizing  this  data  so  that  it  can  be  transmitted  to  a 
ground  station  via  a  radio-frequency  (RF)  link.  The  RF  link  is  implemented  using  a  commercial,  off-the-shelf 
(COTS)  transceiver.  Furthermore,  the  telecommunications  subsystem  is  able  to  receive  uplink  telecommands 
for  controlling  image  acquisition,  vary  the  sampling  frequency  of  individual  sensors,  and  release  the  payload 
from  the  balloon.  The  spacecraft  -  attached  to  a  parachute  -  can  be  tracked  with  global  positioning  system 
(GPS)  data  so  that  it  may  be  safely  recovered  after  its  descent.  The  large-scale  scope  of  this  project,  coupled 
with  the  group  size,  has  led  to  many  new  experiences  for  the  students,  including  an  appreciation  for  true 
teamwork  and  the  positive  and  negative  aspects  of  group  dynamics.  The  efficacy  of  applying  this  systems 
engineering  approach  to  a  variety  of  large-scale  student  projects,  such  as  spacecraft  or  solar-powered  vehicle 
design,  is  also  discussed. 
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The  following  conference  publications/presentations  based  on  this  research  were  supported  in  part  by 
this  ARO  grant: 

Chang-Hee  Won,  Darryl  Sale,  Richard  R.  Schultz,  Arnold  F.  Johnson,  and  William  H.  Semke, 
“Spacecraft  Systems  Engineering  -  The  Initiation  of  a  Multidisciplinary  Design  Project  at  the 
University  of  North  Dakota.”  In  Proceedings  of  the  2001  American  Society  for  Engineering  Education 
Annual  Conference  &  Exposition  (on  CD-ROM),  Electrical  and  Computer  Engineering  Division, 
Albuquerque,  NM,  24-27  Jun  2001.  Presented  at  the  conference  by  Richard  R.  Schultz. 

Christopher  J.  Schmidt,  Joseph  R.  Rydel,  Nicholas  E.  Hulst,  Jonathan  A.  Lovseth,  Darryl  Sale,  Chang- 
Hee  Won,  Richard  R.  Schultz,  Arnold  F.  Johnson,  and  William  H.  Semke,  “Design  of  a  Balloon- 
Launched  Spacecraft.”  Abstract  presented  by  Christopher  J.  Schmidt,  Joseph  R.  Rydel,  and  Jonathan  A. 
Lovseth  at  the  Northern  State  University  Undergraduate  Research  Conference,  Aberdeen,  SD,  20-21  Apr 
2001. 

Christopher  J.  Schmidt,  Jonathan  A.  Lovseth,  Melissa  A.  Barnum,  Jayson  F.  Clairmont,  Patricia  E. 
Langwost,  Nicholas  E.  Hulst,  Kelani  J.  Parisien,  Joseph  R.  Rydel,  Darryl  Sale,  Richard  R.  Schultz, 
Chang-Hee  Won,  Arnold  F.  Johnson,  and  William  H.  Semke,  “Systems  Engineering  Pedagogy  Through 

Balloon-Launched  Spacecraft.”  In  Proceedings  of  the  2001  National  Conference  on  Undergraduate 
Research ,  Lexington,  KY,  15-17  Mar  2001.  Presented  at  the  conference  by  Christopher  J.  Schmidt  and 
Jonathan  A.  Lovseth. 


6.  Listing  of  All  Publications  and  Technical  Reports  Supported  Under  this  Grant  or  Contract 

Provide  the  list  with  the  following  breakout,  and  in  standard  format  showing  authors,  title,  journal, 
issue,  and  date: 

(a)  Papers  published  in  peer-reviewed  journals 

(b)  Papers  published  in  non-peer-reviewed  journals  or  in  conference  proceedings 

(c)  Papers  presented  at  meetings,  but  not  published  in  conference  proceedings 

(d)  Manuscripts  submitted,  but  not  published 

(e)  Technical  reports  submitted  to  ARO 


In  the  following  lists  of  publications,  the  names  of  all  ARO-supported  personnel  are  underlined.  All  lists  are 
in  reverse  chronological  order,  according  to  the  date  of  publication/presentation. 


6(a)  Papers  published  in  peer-reviewed  journals 

1.  Richard  R.  Schultz.  Li  Meng,  and  Robert  L.  Stevenson,  “Subpixel  Motion  Estimation  for  Super- 

Resolution  Image  Sequence  Enhancement,”  Journal  of  Visual  Communication  and  Image  Representation , 
volume  9,  number  1,  pages  38-50,  Mar  1998. 
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6(b)  Papers  published  in  non-peer-reviewed  journals  or  in  conference  proceedings 


1.  Chang-Hee  Won,  Darryl  Sale,  Richard  R.  Schultz,  Arnold  F.  Johnson,  and  William  H.  Semke, 
“Spacecraft  Systems  Engineering  -  The  Initiation  of  a  Multidisciplinary  Design  Project  at  the  University 
of  North  Dakota.”  In  Proceedings  of  the  2001  American  Society  for  Engineering  Education  Annual 
Conference  &  Exposition  (on  CD-ROM),  Electrical  and  Computer  Engineering  Division,  Albuquerque, 
NM,  24-27  Jun  2001.  Presented  at  the  conference  by  Richard  R.  Schultz. 

2.  Richard  R.  Schultz,  “Nonlinear  Filtering  of  Subpixel  Motion  Vectors  for  Improved  Super-Resolution 
Video  Frame  Enhancement.”  Invited  paper  in  Proceedings  of  the  2001 IEEE-EURASIP  Workshop  on 
Nonlinear  Signal  and  Image  Processing  (on  CD-ROM).  Presented  in  the  Special  Session  on  Nonlinear 
Methods  for  Signal  Resolution  Enhancement  (Chairs:  R.  L.  Stevenson  and  R.  Hardie),  Baltimore,  MD,  3- 
6  Jun  2001 .  Presented  at  the  conference  by  Richard  R.  Schultz. 

3.  Christopher  J.  Schmidt,  Jonathan  A.  Lovseth,  Melissa  A.  Bamum,  Jayson  F.  Clairmont,  Patricia  E. 
Langwost,  Nicholas  E.  Hulst,  Kelani  J.  Parisien,  Joseph  R.  Rydel  Darryl  Sale,  Richard  R.  Schultz, 
Chang-Hee  Won,  Arnold  F.  Johnson,  and  William  H.  Semke,  “Systems  Engineering  Pedagogy  Through 
Balloon-Launched  Spacecraft.”  In  Proceedings  of  the  2001  National  Conference  on  Undergraduate 
Research ,  Lexington,  KY,  15-17  Mar  2001 .  Presented  at  the  conference  by  Christopher  J.  Schmidt  and 
Jonathan  A.  Lovseth. 

4.  Darryl  Sale  and  Richard  R.  Schultz,  “Super-Resolution  Enhancement  of  Night  Vision  Image  Sequences.” 
In  Proceedings  of  the  IEEE  Systems,  Man,  and  Cybernetics  Society  2000  Meeting  (on  CD-ROM), 
Nashville,  TN,  8-11  Oct  2000.  Presented  at  the  meeting  by  Richard  R.  Schultz. 

5.  Darryl  Sale,  Richard  R.  Schultz,  Arnold  F.  Johnson,  Trevor  Timpane,  and  William  H.  Semke,  “The 
Initiation  of  a  Satellite  Systems  Engineering  Program  at  the  University  of  North  Dakota.”  In  Proceedings 
of  the  62nd  Annual  American  Society  for  Engineering  Education  North  Midwest  Section  Meeting  (on  CD- 
ROM),  Minneapolis,  MN,  28-30  Sep  2000.  Presented  at  the  meeting  by  Richard  R.  Schultz. 

6.  Kvle  J.  Erickson  and  Richard  R.  Schultz.  “MPEG-1  Super-Resolution  Decoding  for  the  Analysis  of  Video 
Still  Images.”  In  Proceedings  of  the  2000  IEEE  Southwest  Symposium  on  Image  Analysis  and 
Interpretation ,  pages  13-17,  Austin,  TX,  2-4  Apr  2000.  Presented  at  the  symposium  by  Richard  R. 
Schultz. 

7.  Richard  R.  Schultz.  “Experiences  in  the  Integration  of  Digital  Signal  and  Image  Processing  Research  into 
the  Undergraduate  Electrical  Engineering  Curriculum.”  In  Proceedings  of  the  1999  American  Society  for 
Engineering  Education  Annual  Conference  &  Exposition  (on  CD-ROM),  Charlotte,  NC,  20-23  Jun  1999. 
Presented  at  the  conference  by  Richard  R.  Schultz. 

8.  Melissa  L.  Kurtz,  Allison  D.  Sawrey,  Brian  D.  Pedersen,  Mark  J.  Bailey,  Scott  B.  Wing,  Steve  Lindaas, 
Richard  R.  Schultz,  George  Bibel,  Arnold  F.  Johnson,  and  Charles  A.  Wood,  “Design  of  the  Telerobotic 
Adventures  Web-Controlled  Rover  for  the  Dakota  Science  Center’s  Online  Learning  Environment.”  In 
Proceedings  of  the  1999  National  Conference  on  Undergraduate  Research ,  Rochester,  NY,  8-10  Apr 
1999.  Presented  at  the  conference  by  Melissa  L.  Kurtz. 

9.  Richard  R.  Schultz  and  Mark  G.  Alford,  “Automated  Image  Registration  Using  the  Projective 
Transformation  Model  and  Block  Matching  Feature  Point  Pair  Selection.”  In  Proceedings  of  SPIE  - 
Sensor  Fusion:  Architectures,  Algorithms,  and  Applications  III  (AeroSense  ‘99),  volume  3719,  pages 
140-151,  Orlando,  FL,  7-9  Apr  1999.  Presented  at  the  conference  by  Richard  R.  Schultz. 

10.  Richard  R.  Schultz  and  Mark  G.  Alford,  “Multiframe  Integration  via  the  Projective  Transformation  with 
Automated  Block  Matching  Feature  Point  Selection.”  In  Proceedings  of  the  1999  IEEE  International 
Conference  on  Acoustics,  Speech,  and  Signal  Processing  (on  CD-ROM),  Phoenix,  AZ,  15-19  Mar  1999. 
Presented  at  the  conference  by  Richard  R.  Schultz. 

1 1 .  Debin  Chen  and  Richard  R.  Schultz.  “Extraction  of  High-Resolution  Frames  from  MPEG  Image 
Sequences.”  In  Proceedings  of  the  1998  IEEE  International  Conference  on  Image  Processing  (on  CD- 
ROM),  Chicago,  IL,  4-7  Oct  1998.  Presented  at  the  conference  by  Richard  R.  Schultz. 
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12.  Kyle  J.  Erickson  and  Richard  R.  Schultz,  “A  Windows-Based  Software  Tool  for  the  Super-Resolution 
Enhancement  of  Surveillance  Image  Sequences.”  In  Proceedings  of  the  1998  Midwest  Symposium  on 
Circuits  and  Systems ,  pages  379-382,  Notre  Dame,  IN,  9-12  Aug  1998.  Presented  at  the  symposium  by 
Kyle  J.  Erickson. 

13.  Kevin  G.  Rada,  Richard  R.  Schultz,  and  Jody  A.  Rada,  “In  Vivo  Depth  Estimation  of  Features  in  the 
Chick  Eye  by  Digital  Signal  Processing  of  Ultrasonic  Pulse  Echoes.”  In  Proceedings  of  the  1998 
Midwest  Symposium  on  Circuits  and  Systems ,  pages  326-329,  Notre  Dame,  IN,  9-12  Aug  1998. 

Presented  at  the  symposium  by  Kevin  G.  Rada. 

14.  Richard  R.  Schultz.  “A  Practical  Introduction  to  Digital  Signal  Processing  through  Microsoft  Visual  C++ 
and  Lab  VIEW  Programming.”  In  Proceedings  of  the  1998  American  Society  for  Engineering  Education 
Annual  Conference  (on  CD-ROM),  Seattle,  WA,  28  Jun  1998  through  1  Jul  1998.  Presented  at  the 
conference  by  Richard  R.  Schultz. 

15.  Michael  J.  Carman,  Aaron  Douglas,  Dawn  C.  Willett,  and  Richard  R.  Schultz,  “Multispectral 
Segmentation  of  the  Visible  Human  Project  Data  Set  via  Vector  Quantization.”  In  Proceedings  of  the 
1998  National  Conference  on  Undergraduate  Research ,  Salisbury,  MD,  23-25  Apr  1998.  Not  presented 
at  the  conference  because  of  time  constraints  on  personnel. 

16.  Richard  R.  Schultz  and  Robert  L.  Stevenson,  “Estimation  of  Subpixel-Resolution  Motion  Fields  from 
Segmented  Image  Sequences.”  In  Proceedings  of  SPIE  -  Sensor  Fusion:  Architectures,  Algorithms,  and 
Applications  II  (AeroSense  ‘98),  volume  3376,  pages  90-101,  Orlando,  FL,  13-17  Apr  1998.  Presented  at 
the  conference  by  Richard  R.  Schultz. 

17.  Richard  R.  Schultz  and  Robert  L.  Stevenson,  “Bayesian  Estimation  of  Subpixel-Resolution  Motion  Fields 
and  High-Resolution  Video  Stills.”  In  Proceedings  of  the  1997  IEEE  International  Conference  on  Image 
Processing  (on  CD-ROM),  Santa  Barbara,  CA,  26-29  Oct  1997.  Presented  at  the  conference  by  Richard 
R.  Schultz. 

18.  Richard  R.  Schultz,  “Teaching  Signals  and  Systems  through  Visualization  with  Image  Processing.”  In 
Proceedings  of  the  1997  American  Society  for  Engineering  Education  Annual  Conference  (on  CD-ROM), 
Milwaukee,  WI,  15-18  Jun  1997.  Presented  at  the  conference  by  Richard  R.  Schultz. 

19.  Melissa  L.  Kurtz  and  Richard  R.  Schultz,  “Nonlinear  Filtering  and  Enhancement  of  Digital  Ultrasound 
Imagery.”  In  Proceedings  of  the  1997  National  Conference  on  Undergraduate  Research ,  Austin,  TX,  24- 
26  Apr  1997.  Presented  at  the  conference  by  Melissa  L.  Kurtz. 

20.  James  A.  Johnson  and  Richard  R.  Schultz,  “SubzeroCam:  Design  of  a  World  Wide  Web-Based  Digital 
Camera  for  Sunrayce  ‘97.”  In  Proceedings  of  the  1997  National  Conference  on  Undergraduate 
Research ,  Austin,  TX,  24-26  Apr  1997.  Presented  at  the  conference  by  Melissa  L.  Kurtz  (proxy). 

21 .  Richard  R.  Schultz,  “A  Hybrid  Block  Matching  and  Optical  Flow  Equation-Based  Motion  Estimation 
Algorithm  for  Digital  Video  Sequences.”  In  Proceedings  of  the  ISCA  12th  International  Conference  on 
Computers  and  Their  Applications,  pages  74-77,  Tempe,  AZ,  13-15  Mar  1997.  Presented  at  the 
conference  by  Richard  R.  Schultz. 

22.  Richard  R.  Schultz,  Li  Meng,  and  Robert  L.  Stevenson,  “Subpixel  Motion  Estimation  for  Multiframe 
Resolution  Enhancement.”  In  Proceedings  of  Visual  Communications  and  Image  Processing  ‘97,  volume 
3024,  pages  1317-1328,  San  Jose,  CA,  Feb  1997.  Presented  at  the  conference  by  Richard  R.  Schultz. 
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6(c)  Papers  presented  at  meetings,  but  not  published  in  conference  proceedings 

1.  Christopher  J.  Schmidt,  Joseph  R.  Rydel  Nicholas  E.  Hulst,  Jonathan  A.  Lovseth,  Darryl  Sale,  Chang- 
Hee  Won,  Richard  R.  Schultz,  Arnold  F.  Johnson,  and  William  H.  Semke,  “Design  of  a  Balloon- 
Launched  Spacecraft.”  Abstract  presented  by  Christopher  J.  Schmidt,  Joseph  R.  Rydel,  and  Jonathan  A. 
Lovseth  at  the  Northern  State  University  Undergraduate  Research  Conference,  Aberdeen,  SD,  20-21  Apr 
2001. 

2.  Kevin  G.  Rada,  Jody  A.  Rada,  and  Richard  R.  Schultz,  “An  Automated  Tissue  Boundary  Detection 
Algorithm  for  Ultrasonic  Intraocular  Studies  Based  on  Digital  Matched  Filtering.”  Abstract  presented  by 
Kevin  G.  Rada  at  the  1999  Association  for  Research  in  Vision  and  Ophthalmology  (ARVO)  Annual 
Meeting,  Fort  Lauderdale,  FL,  9-14  May  1999. 


6(d)  Manuscripts  submitted,  but  not  published 

None. 


6(e)  Technical  reports  submitted  to  ARO 

None. 


7.  List  of  All  Participating  Scientific  Personnel 


Mr.  Bradley  R.  Bender,  Graduate  Research  Assistant 

B.S.E.E.:  University  of  North  Dakota,  May  1998 

ARO  Project:  “Digital  Microscopy  Image  Analysis” 

ARO  Support:  Summer  1998 

Mr.  Bender  was  planning  on  attending  graduate  school  at  the  University  of  North  Dakota  to  work  on  this  ARO 
project,  but  he  decided  to  become  a  systems  administrator  within  industry  during  the  summer  of  1998. 


Mr.  Kyle  J.  Erickson,  Graduate  Research  Assistant 

M.S.E.E.:  University  of  North  Dakota,  May  2000 

B.S.C.S.:  University  of  North  Dakota,  August  1998  (Computer  Science) 

Master’s  Thesis:  Super-Resolution  Enhancement  of  MPEG  Compressed  Video 

ARO  Projects:  “Super-Resolution  Enhancement  of  Compressed  Digital  Video”  and  “Development  of  a 

Windows-Based  Software  Tool  for  the  Super-Resolution  Enhancement  of  Surveillance 
Image  Sequences” 

ARO  Support:  Fall  1998,  Spring  1999,  and  Summer  1999 


Mr.  Brian  G.  Giesinger,  Undergraduate  Research  Assistant 

M.S.E.E.  (Expected):  University  of  North  Dakota,  May  2002 

B.S.E.E.:  University  of  North  Dakota,  May  2000 

Master’s  Thesis:  Design  of  an  Imaging  System  for  AgCam  -  The  International  Space  Station-Based  Earth 

Observing  Camera  for  Precision  Agriculture 

ARO  Projects:  “Design  of  a  Weather-Balloon  Launched  Spacecraft”  and  “Digital  Microscopy  Image 

Analysis” 

ARO  Support:  Summer  1998  and  Summer  1999 

“AgCam”  is  a  multidisciplinary  research  project  between  UND  Engineering  and  the  Upper  Midwest  Aerospace 
Consortium  (UMAC),  based  at  the  University  of  North  Dakota. 
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Mr.  Nicholas  E.  Hulst,  Undergraduate  Research  Assistant 

B.S.E.E.  (Expected):  University  of  North  Dakota,  May  2002 

ARO  Project:  “Design  of  a  Weather-Balloon  Launched  Spacecraft” 

ARO  Support:  Summer  2000 

Mr.  Hulst  has  been  accepted  to  attend  graduate  school  at  the  University  of  North  Dakota  starting  in  August  2002. 
He  plans  to  serve  as  a  graduate  research  assistant  on  the  “AgCam”  project. 


Mr.  Joseph  R.  Rydel,  Undergraduate  Research  Assistant 

B.S.E.E.:  University  of  North  Dakota,  May  2001 

ARO  Projects:  “Design  of  a  Weather-Balloon  Launched  Spacecraft”  and  “Design  of  a  World  Wide  Web 

Interface  for  the  UND  Space  Studies  Telescope” 

ARO  Support:  Summer  1998  and  Summer  2000 


Mr.  Darryl  L.  Sale,  Research  Engineer 

ARO  Projects:  “Design  of  a  Weather-Balloon  Launched  Spacecraft,”  “Super-Resolution  Enhancement 

of  Night  Vision  Image  Sequences,”  and  “Super-Resolution  Enhancement  of  Compressed 
Digital  Video” 

ARO  Support:  Spring  2000,  Summer  2000,  Fall  2000,  Spring  2001,  and  Summer  2001 


Dr.  Richard  R.  Schultz,  Principal  Investigator 

Role  in  ARO  Grant:  Richard  R.  Schultz  served  as  the  P.I.  of  this  ARO  grant,  supervising  all  project  activities 

and  documenting  their  results. 

ARO  Salary  Support:  Summer  1999,  Summer  2000,  and  Summer  2001 


Ms.  Nicole  A.  Turner,  Undergraduate  Research  Assistant 

B.S.E.E.:  University  of  North  Dakota,  May  2000 

ARO  Project:  “Digital  Microscopy  Image  Analysis” 

ARO  Support:  Summer  1999 

Ms.  Turner  is  currently  attending  graduate  school  at  North  Dakota  State  University  (Fargo,  ND)  in  electrical 
engineering,  perfonning  research  in  biomedical  and  cardiovascular  engineering. 


8.  Report  of  Inventions 
Patent  Application 


Patent  Title: 
Inventors: 
Assignee: 
Filing  Date: 
Serial  Number: 
Law  Finn: 
Attorney: 


“Super-Resolution  Enhancement  of  Compressed  Video  Data” 

Richard  R.  Schultz  and  Kyle  J.  Erickson 

University  of  North  Dakota 

12  May  2000 

60/133,765 

Shook,  Hardy  &  Bacon  L.L.P.  (Kansas  City,  MO) 

Mr.  Michael  J.  Gross 
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